Dropout regularization in hierarchical mixture of experts

Dropout is a very effective method in preventing overfitting and has become the go-to regularizer for multi-layer neural networks recent years. Hierarchical mixture of experts hierarchically gated model that defines soft decision tree where leaves correspond to nodes gating models softly choose between its children, as such, hierarchical partitioning input space. In this work, we propose variant dropout faithful hierarchy defined by model, opposed having flat, unitwise independent application one with perceptrons. We show on synthetic regression data MNIST, CIFAR-10, SSTB datasets, our proposed mechanism prevents trees many levels improving generalization providing smoother fits.